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Abstract 



Coecke, Sadrzadeh, and Clark [3 | developed a compositional model of meaning for distributional 
semantics, in which each word in a sentence has a meaning vector and the distributional meaning of the 
sentence is a function of the tensor products of the word vectors. Abstractly speaking, this function is the 
morphism corresponding to the grammatical structure of the sentence in the category of finite dimensional 
vector spaces. In this paper, we provide a concrete method for implementing this linear meaning map, 
■ by constructing a corpus-based vector space for the type of sentence. Our construction method is based 

, on structured vector spaces whereby meaning vectors of all sentences, regardless of their grammatical 

^ ■ structure, live in the same vector space. Our proposed sentence space is the tensor product of two noun 

spaces, in which the basis vectors are pairs of words each augmented with a grammatical role. This 
enables us to compare meanings of sentences by simply taking the inner product of their vectors. 

> 
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Coecke, Sadrzadeh, and Clark [3] develop a mathematical framework for a compositional distributional 
model of meaning, based on the intuition that syntactic analysis guides the semantic vector composition. 
The setting consists of two parts: a formalism for a type-logical syntax and a formalism for vector space 
semantics. Each word is assigned a grammatical type and a meaning vector in the space corresponding to 
its type. The meaning of a sentence is obtained by applying the function corresponding to the grammatical 
structure of the sentence to the tensor product of the meanings of the words in the sentence. Based on the 
type-logic used, some words will have atomic types and some compound function types. The compound 
types live in a tensor space where the vectors are weighted sums (i.e. superpositions) of the pairs of bases 
from each space. Compound types are "applied" to their arguments by taking inner products, in a similar 
manner to how predicates are applied to their arguments in Montague semantics. 

For the type-logic we use Lambek's Pregroup grammars Q. The use of pregoups is not essential, but 
leads to a more elegant formalism, given its proximity to the categorical structure of vector spaces (see O). 
A Pregroup is a partially ordered monoid where each element has a right and left cancelling element, referred 
to as an adjoint. It can be seen as the algebraic counterpart of the cancellation calculus of Harris I0. The 
operational difference between a Pregroup and Lambek's Syntactic Calculus is that, in the latter, the monoid 
multiplication of the algebra (used to model juxtaposition of the types of the words) has a right and a left 
adjoint, whereas in the pregroup it is the elements themselves which have adjoints. The adjoint types are 
used to denote functions, e.g. that of a transitive verb with a subject and object as input and a sentence as 
output. In the Pregroup setting, these function types are still denoted by adjoints, but this time the adjoints 
of the elements themselves. 

As an example, consider the sentence "dogs chase cats". We assign the type n (for noun phrase) to "dog" 
and "cat", and n r sn l to "chase", where n r and n l are the right and left adjoints of n and s is the type of a 



(declarative) sentence. The type n r sn l expresses the fact that the verb is a predicate that takes two arguments 
of type n as input, on its right and left, and outputs the type s of a sentence. The parsing of the sentence is 
the following reduction: 

n(n r sn l )n < lsl = s 

This parse is based on the cancellation of n and n r , and also n l and n; i.e. nrf < 1 and n l n < 1 for 1 
the unit of juxtaposition. The reduction expresses the fact that the juxtapositions of the types of the words 
reduce to the type of a sentence. 

On the semantic side, we assign the vector space N to the type n, and the tensor space 2V ® S ® N to the 
type n r sn l . Very briefly, and in order to introduce some notation, recall that the tensor space A®B has as a 
basis the cartesian product of a basis of A with a basis of B. Recall also that any vector can be expressed as 
a weighted sum of basis vectors; e.g. if (yi, . . . , v n ) is a basis of A then any vector a & A can be written as 

~ct = J2i Civt where each Q E R is a weighting factor. Now for (vt, . . . , v%) a basis of A and (y\, . . . , v^) 
a basis of B, a vector c in the tensor space A® B can be expressed as follows: 



E 



Cij (vt ® Vj) 



where the tensor of basis vectors vt ® v^ stands for their pair (vt, v^). In general if is not separable into 
the tensor of two vectors, except for the case when it is not entangled. For non-entangled vectors we can 
write it = it ® b for it = ]TV d vt and b = ^ • Cjt^; hence the weighting factor of 1? can be obtained 
by simply multiplying the weights of its tensored counterparts, i.e. Cij = Ci x Cj. In the entangled case 
these weights cannot be determined as such and range over all the possibilities. We take advantage of this 
fact to encode meanings of verbs, and in general all words that have compound types and are interpreted as 
predicates, relations, or functions. For a brief discussion see the last paragraph of this section. Finally, we 
use the Dirac notation to denote the dot or inner product of two vectors (a \ b } € R defined by Yli Ci x C[. 



Returning to our example, for the meanings of nouns we have dogs, cats G N, and for the meanings of 
verbs we have chase £ N <g> S <S> N, i.e. the following superposition: 



E 



Here and n£ are basis vectors of N and Sj is a basis vector of S. From the categorical translation method 
presented in and the grammatical reduction n(n r sn l )n < s, we obtain the following linear map as the 
categorical morphism corresponding to the reduction: 

e N <g> l s ® e N : N <g> (N <g> S ® N) ® AT -> 5 

Using this map, the meaning of the sentence is computed as follows: 



dogs chase cats = (e^ ® l s ® eAr) I dogs ® chase ® cats I 



(eN ® l s ® e^) ^dogs ® ^2 Cijk(r% ® s| ® ® cats^ 
y^Cijfc(dogs I n-)i}(nt | cats) 



The key features of this operation are, first, that the inner-products reduce dimensionality by 'consuming' 
tensored vectors and by virtue of the following component function: 



e A r:7V®A^^R::o f ® 6 ^ ( | 6) 



Thus the tensored word vectors dogs ® chase & cats are mapped into a sentence space S which is common 
to all sentences regardless of their grammatical structure or complexity. Second, note that the tensor product 
dogs ® chase (g) cats does not need to be calculated, since all that is required for computation of the sentence 
vector are the noun vectors and the Cyk weights for the verb. Note also that the inner product operations 
are simply picking out basis vectors in the noun space, an operation that can be performed in constant 
time. Hence this formalism avoids two problems faced by approaches in the vein of (9j 13, which use 
the tensor product as a composition operation: first, that the sentence meaning space is high dimensional 
and grammatically different sentences have representations with different dimensionalities, preventing them 
from being compared directly using inner products; and second, that the space complexity of the tensored 
representation grows exponentially with the length and grammatical complexity of the sentence. In constrast, 
the model we propose does not require the tensored vectors being combined to be represented explicitly. 

Note that we have taken the vector of the transitive verb, e.g. chase, to be an entangled vector in the 
tensor space N ® S <g> N. But why can this not be a separable vector, in which case the meaning of the verb 
would be as follows: 

chase = ^Qr^ ® ^Cjs) ® ^ C% n£ 

i j k 

The meaning of the sentence would then become o\02 J2j Cj'sj for o\ = J2i Cj(dogs | n\) and 02 = 
Cheats I nt). The problem is that this meaning only depends on the meaning of the verb and is 
independent of the meanings of the subject and object, whereas the meaning from the entangled case, 
i.e. (7\(72 Ylijk Cjjfc^l' depends on the meanings of subject and object as well as the verb. 



2 From Truth-Theoretic to Corpus-based Meaning 

The model presented above is compositional and distributional, but still abstract. To make it concrete, N and 
S have to be constructed by providing a method for determining the Cijk weightings. Coecke, Sadrzadeh, 
and Clark [3] show how a truth-theoretic meaning can be derived in the compositional framework. For 
example, assume that N is spanned by all animals and S is the two-dimensional space spanned by true and 
false. We use the weighting factor to define a model-theoretic meaning for the verb as follows: 

_^ Jtrue chase{n%, n£) = true , 
I false o.w. 

The definition of our meaning map ensures that this value propagates to the meaning of the whole sentence. 
So chase{dogs, cats) becomes true whenever "dogs chase cats" is true and false otherwise. This is exactly 
how meaning is computed in the model-theoretic view on semantics. One way to generalise this truth- 
theoretic meaning is to assume that chaseijii , n%) has degrees of truth, for instance by defining chase as a 
combination of run and catch, such as: 

2 1 
chase = -run H — catch 

3 3 

Again, the meaning map ensures that these degrees propagate to the meaning of the whole sentence. For a 
worked out example see |3 ]. But neither of these examples provide a distributional sentence meaning. 

Here we take a first step towards a corpus-based distributional model, by attempting to recover a meaning 
for a sentence based on the meanings of the words derived from a corpus. But crucially this meaning goes 
beyond just composing the meanings of words using a vector operator, such as tensor product, summation 
or multiplication flSJ. Our computation of sentence meaning treats some vectors as functions and others as 



function arguments, according to how the words in the sentence are typed, and uses the syntactic structure 
as a guide to determine how the functions are applied to their arguments. The intuition behind this approach 
is that syntactic analysis guides semantic vector composition. 

The contribution of this paper is to introduce some concrete constructions for a compositional distri- 
butional model of meaning. These constructions demonstrate how the mathematical model of can be 
implemented in a concrete setting which introduces a richer, not necessarily truth- theoretic, notion of natural 
language semantics which is closer to the ideas underlying standard distributional models of word meaning. 
We leave full evaluation to future work, in order to determine whether the following method in conjunction 
with word vectors built from large corpora leads to improved results on language processing tasks, such as 
computing sentence similarity and paraphrase evaluation. 

Nouns and Transitive Verbs. We take N to be a structured vector space, as in Hill. The bases of N are 
annotated by 'properties' obtained by combining dependency relations with nouns, verbs and adjectives. For 
example, basis vectors might be associated with properties such as "arg-fluffy", denoting the argument of 
the adjective fluffy, "subj-chase" denoting the subject of the verb chase, "obj-buy" denoting the object of the 
verb buy, and so on. We construct the vector for a noun by counting how many times in the corpus a word 
has been the argument of 'fluffy', the subject of 'chase', the object of 'buy', and so on. 

The framework in [ 3 ] offers no guidance as to what the sentence space should consist of. Here we take 
the sentence space S to be N ® N, so its bases are of the form ~s~j = (n^ , n&). The intuition is that, for a 
transitive verb, the meaning of a sentence is determined by the meaning of the verb together with its subject 
and objectQ The verb vectors C^ir^i, n£) are built by counting how many times a word that is m (e.g. has 
the property of being fluffy) has been subject of the verb and a word that is n& (e.g. has the property that it's 
bought) has been its object, where the counts are moderated by the extent to which the subject and object 
exemplify each property (e.g. how fluffy the subject is). To give a rough paraphrase of the intuition behind 
this approach, the meaning of "dog chases cat" is given by: the extent to which a dog is fluffy and a cat is 
something that is bought (for the N (g> N property pair "arg-fluffy" and "obj-buy"), and the extent to which 
fluffy things chase things that are bought (accounting for the meaning of the verb for this particular property 
pair); plus the extent to which a dog is something that runs and a cat is something that is cute (for the A <g> A 
pair "subj-run" and "arg-cute"), and the extent to which things that run chase things that are cute (accounting 
for the meaning of the verb for this particular property pair); and so on for all noun property pairs. 

Adjective Phrases. Adjectives are dealt with in a similar way. We give them the syntactic type nn and 
build their vectors in A ® A. The syntactic reduction nn l n — > n associated with applying an adjective to a 
noun gives us the map ljy <8> (-N by which we semantically compose an adjective with a noun, as follows: 



We can view the C{j counts as determining what sorts of properties the arguments of a particular adjective 
typically have (e.g. arg-red, arg-colourful for the adjective "red"). 

Prepositional Phrases. We assign the type n r n to the whole prepositional phrase (when it modifies a noun), 
for example to "in the forest" in the sentence "dogs chase cats in the forest". The pregroup parsing is as 
follows: 



The vector space corresponding to the prepositional phrase will thus be the tensor space A A and the 
categorification of the parse will be the composition of two morphisms: (ls®e^)o(e^®ls®lAr®e^Cg)l/v')- 




n(n r sn )n(n r n) < lsn In < sn n < si = s 



'intransitive and di transitive verbs are interpreted in an analagous fashion; see ij4] 



The substitution specific to the prepositional phrase happens when computing the vector for "cats in the 
forest" as follows: 



Here we set the weights C\ w in a similar manner to the cases of adjective phrases and verbs with the counts 
determining what sorts of properties the noun modified by the prepositional phrase has, e.g. the number of 
times something that has attribute ni has been in the forest. 

Adverbs. We assign the type s r s to the adverb, for example to "quickly" in the sentence "Dogs chase cats 
quickly". The pregroup parsing is as follows: 



Its categorification will be a composition of two morphisms (e r s <g> Is) ° (eyv ® Is ® % ® Is ® Is)- The 
substitution specific to the adverb happens after computing the meaning of the sentence without it, i.e. that 
of "Dogs chase cats", and is as follows: 



Dogs chase cats quickly = (e r s <8> Is) ° (e^v ® Is ® ® Is ® Is) ( Dogs ® chase <g> cats <g) quickly 



The C\ w weights are defined in a similar manner to the above cases, i.e. according to the properties the 
adverb has, e.g. which verbs it has modified. Note that now the basis vectors s| and are themselves pairs 
of basis vectors from the noun space, (n^, nj). Hence, Ci w (r%, nj) can be set only for the case when I = i 
and w = j; these counts determine what sorts of properties the verbs that happen quickly have (or more 
specifically what properties the subjects and objects of such verbs have). By taking the whole sentence into 
account in the interpretation of the adverb, we are in a better position to semantically distinguish between 
the meaning of adverbs such as "slowly" and "quickly", for instance in terms of the properties that the verb's 
subjects have. For example, it is possible that elephants are more likely to be the subject of a verb which is 
happening slowly, e.g. run slowly, and cheetahs are more likely to be the subject of a verb which is happening 
quickly. 



cats in the forest = (e r N ® ljv) (cats <g> in the forest 




Iw 



n(n r sn )n(s r s) < lsls r s = ss r s < Is = s 




3 Concrete Computations 



In this section we first describe how to obtain the relevant counts from a parsed corpus, and then give some 
similarity calculations for some example sentence pairs. 



Let Ci be the set of grammatical relations (GRs) for sentence si in the corpus. Define verbs(Ci) to be 
the function which returns all instances of verbs in C\, and subj (and similarly obj) to be the function which 
returns the subject of an instance Vi nstance of a verb V, for a particular set of GRs for a sentence: 





Y.veverbsid) V)(subj(v) \ 7%){obj(v) I 4) KSj = (4, 4) 



where S(v, V) = 1 if t> = F and otherwise. Thus we construct for verb V only for cases where 
the subject property rii and the object property nk are paired in the basis i|. This is done by counting the 
number of times the subject of V has property n« and the object of V has property then multiplying them, 
as prescribed by the inner products (which simply pick out the properties rii and re& from the noun vectors 
for the subjects and objects). 

The procedure for calculating the verb vectors, based on the formulation above, is as follows: 

1. For each GR in a sentence, if the relation is subject and the head is a verb, then find the complementary 
GR with object as a relation and the same head verb. If none, set the object to e n . 

> > 

2. Retrieve the noun vectors subject, object for the subject dependent and object dependent from previ- 
ously constructed noun vectors. 

3. For each (rii, n&) € basis(N) x basis(N) compute the inner-product of nt with subject and nt with 
object (which involves simply picking out the relevant basis vectors from the noun vectors). Multiply 
the inner-products and add this to for the verb, with j such that s| = (nt, ret). 

The procedure for other grammatical types is similar, based on the definitions of C weights for the semantics 
of these types. 

We now give a number of example calculations. We first manually define the distributions for nouns, 
which in practice would be obtained from a corpus: 



We aim to make these counts match our intuitions, in that bankers are shrewd and a little ferocious but not 
furry, cats are furry but not typically valuable, and so on. 

We also define the distributions for the transitive verbs 'chase', 'pursue' and 'sell', again manually 
specified according to our intuitions about how these verbs are used. Since in the formalism proposed above, 
Cijk = if ~s~j 7^ (nt, nt), we can simplify the weight matrices for transitive verbs to two dimensional 
matrices as shown below, where corresponds to the number of times the verb has a subject with attribute 
rii and an object with attribute n^. For example, the matrix below encodes the fact that something ferocious 



bankers cats dogs stock kittens 



1. arg-fluffy 



7 3 2 

4 16 

4 2 7 

6 3 10 1 

12 8 



2. arg-ferocious 

3. obj-buys 



4. arg-shrewd 

5. arg-valuable 



(i = 2) chases something fluffy (k = 1) seven times in the hypothetical corpus from which we might have 
obtained these distributions. 



C 



chase 



1 
7 12 3 1 



2 10 1 

1 



^rpursue 





4 2 2 2 4 



3 2 1 





These matrices can be used to perform sentence comparisons: 



(dogs chase cats | dogs pursue kittens) = 



^C^ se (dogs|nt)5j(^|cats) 

ijk 



c 



sell 





3 4 



5 8 

10 1 



Cf" fc rsue (dogs | ni)sj(nk | kittens) 



= E ^r^Sr<dogs | 4) (dogs | n>><4 I cats)<njt I kiHe^) 

ijk 

The raw number obtained from the above calculation is 14844. Normalising it by the product of the length 

of both sentence vectors gives the cosine value of 0.979. 

> > 

Consider now the sentence comparison (dogs chase cats | cats chase dogs). The sentences in this pair 

contain the same words but the different word orders give the sentences very different meanings. The raw 
number calculated from this inner product is 7341, and its normalised cosine measure is 0.656, which demon- 
strates the sharp drop in similarity obtained from changing sentence structure. We expect some similarity 
since there is some non-trivial overlap between the properties identifying cats and those identifying dogs 
(namely those salient to the act of chasing). 

Our final example for transitive sentences is (dogs chase cats | bankers sell stock), as two sentences that 
diverge in meaning completely. The raw number for this inner product is 6024, and its cosine measure is 
0.042, demonstrating the very low semantic similarity between these two sentences. 

Next we consider some examples involving adjective-noun modification. The Cij counts for an adjective 
A are obtained in a similar manner to transitive or intransitive verbs: 



c = J Ei Eoeo^G) 5 ( a > A)(arg-of(a) \ m) if n { 
' o.w. 



where adjs(Ci) returns all instances of adjectives in Cf, 5(a,A) = 1 if a = A and otherwise; and 
arg-of(a) = noun if a is an adjective with argument noun, and e n otherwise. 

As before, we stipulate the CV, matrices by hand (and we eliminate all cases where i ^ j since = 
by definition in such cases): 

^fluffy = [ 9 3 4 2 2] C shrewd = [03191] c*™iuabie = [3 o 8 1 8] 

We compute vectors for "fluffy dog" and "shrewd banker" as follows: 



fluffy dog = (3 • 9) arg-fluffy + (6 • 3) arg-ferocious + (2 • 4) obj-buys + (5 • 2) arg-shrewd + (2 • 2) arg-valuable 



shrewd banker = (0 • 0) arg-fluffy + (4 • 3) arg-ferocious + (0 • 0) obj-buys + (6 • 9) arg-shrewd + (0 • 1) arg-valuable 



. — — r~. ^ 



Vectors for fluffy cat and valuable stock are computed similarly. We obtain the following similarity mea- 
sures: 

> 7—K „ _ > —. — T~. ?\ 



cosine (fluffy dog, shrewd banker) = 0.389 cosme(fluffy cat, valuable stock) = 0.184 



These calculations carry over to sentences which contain the adjective-noun pairings compositionally and 
we obtain an even lower similarity measure between sentences: 

cosine (fluffy dogs chase fluffy cats, shrewd bankers sell valuable stoci) = 0.016 
To summarise, our example vectors provide us with the following similarity measures: 



Sentence 1 


Sentence 2 


Degree of similarity 


dogs chase cats 


dogs pursue kittens 


0.979 


dogs chase cats 


cats chase dogs 


0.656 


dogs chase cats 


bankers sell stock 


0.042 


fluffy dogs chase fluffy cats 


shrewd bankers sell valuable stock 


0.016 



4 Different Grammatical Structures 

So far we have only presented the treatment of sentences with transitive verbs. For sentences with intransitive 
verbs, the sentence space suffices to be just N. To compare the meaning of a transitive sentence with an 
intransitive one, we embed the meaning of the latter from N into the former N <g> N, by taking e% (the 
'object' of an intransitive verb) to be J2i n i-> i- e - tne superposition of all basis vectors of N. 

Following the method for the transitive verb, we calculate for an instransitive verb V and basis pair 
~Sj = (r^i, nt) as follows, where I ranges over the sentences in the corpus: 

E E S{v,V)( S ubj{v) | ^(objtfj | 4) = E E S(v,V)(subj(v)\rt)(I>\4) 

l vGverbs(Ci) I vGverbs(Ci) 

and (e^ | n$) = 1 for any basis vector rij. 

We can now compare the meanings of transitive and intransitive sentences by taking the inner product of 
their meanings (despite the different arities of the verbs) and then normalising it by vector length to obtain 
the cosine measure. For example: 



(dogs chase cats | dogs chase) = / | E Cjjk (dogs | r?i)sj(nk \ cats ) 



ijk 



ijk 



C'ij fc(dogs 



m)sj 



= ^2C ijk C' ijk (dogs | n-)(dogs | n-)(nt I cats) 

ijk 

The raw number for the inner product is 14092 and its normalised cosine measure is 0.961, indicating high 
similarity (but some difference) between a sentence with a transitive verb and one where the subject remains 
the same, but the verb is used intransitively. 

Comparing sentences containing nouns modified by adjectives to sentences with unmodified nouns is straight- 
forward: 

(fluffy dogs chase fluffy cats | dogs chase cats) = 



^ c flufty c flufty c cha S e c chase^ | | ^2 = 3437005 

ij 



From the above we obtain the following similarity measure: 

cosine (fluffy dogs chase fluffy cats, dogs chase cats) = 0.971 

For sentences with ditransitive verbs, the sentence space changes to N <g> N cg> N, on the basis of the verb 
needing two objects; hence its grammatical type changes to n r sn l n l . The transitive and intransitive verbs 
are embedded in this larger space in a similar manner to that described above; hence comparison of their 
meanings becomes possible. 

5 Ambiguous Words 

The two different meanings of a word can be distinguished by the different properties that they have. These 
properties are reflected in the corpus, by the different contexts in which the words appear. Consider the 
following example from [4]: the verb "catch" has two different meanings, "grab" and "contract". They are 
reflected in the two sentences "catch a ball" and "catch a disease". The compositional feature of our meaning 
computation enables us to realise the different properties of the context words via the grammatical roles they 
take in the corpus. For instance, the word 'ball' occurs as argument of 'round', and so has a high weight 
for the base 'arg-round', whereas the word 'disease' has a high weight for the base 'arg-contagious' and as 
'mod-of-heart'. We extend our example corpus from previously to reflect these differences as follows: 





ball 


disease 


1. 


arg-fluffy 


1 





2. 


arg-ferocious 








3. 


obj-buys 


5 





4. 


arg-shrewd 








5. 


arg-valuable 


1 





6. 


arg-round 


8 





7. 


arg-contagious 





7 


8. 


mod-of-heart 





6 



In a similar way, we build a matrix for the verb 'catch' as follows: 

3 3 3 8 6 2 " 
3 1 4 7 4 
7 116 2 2 
2 3 6 2 
1 2 1 


. 

The last three rows are zero because we have assumed that the words that can take these roles are mostly 
objects and hence cannot catch anything. Given these values, we compute the similarity measure between 
the two sentences "dogs catch a ball" and "dogs catch a disease" as follows: 

(dogs catch a ball | dogs catch a disease) = 

In an idealised case like this where there is very little (or no) overlap between the properties of the objects 
associated with one sense of "catch" (e.g. a disease), and those properties of the objects associated with an- 
other sense (e.g. a ball), disambiguation is perfect in that there is no similarity between the resulting phrases. 



C 



catch 



3 


2 


3 


2 


2 


4 


3 


1 


1 


1 





















In practice, in richer vector spaces, we would expect even diseases and balls to share some properties. How- 
ever, as long as those shared properties are not those typically held by the object of catch, and as long as the 
usages of catch play to distinctive properties of diseases and balls, disambiguation will occur by the same 
mechanism as the idealised case above, and we can expect low similarity measures between such sentences. 

6 Related Work 

Mitchell and Lapata introduce and evaluate a multiplicative model for vector composition [8 ]. The particular 
concrete construction of this paper differs from that of (H in that our framework subsumes truth-theoretic 
as well as corpus-based meaning, and our meaning construction relies on and is guided by the grammatical 
structure of the sentence. The approach of is more in the spirit of ours, in that extra information about 
syntax is used to compose meaning. Similar to us, they use a structured vector space to integrate lexical 
information with selectional preferences. Finally, Baroni and Zamparelli model adjective-noun combinations 
by treating an adjective as a function from noun space to noun space, represented using a matrix, as we do 
in this paper fill . 
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